Occlusion in collaborative augmented environments

Augmented environments superimpose computer enhancements on the real world. Such augmented environments are well suited for collaboration of multiple users. To improve the quality and consistency of the augmentation the occlusion of real objects by computergenerated objects and vice versa has to be implemented. We present methods how this can be done for a tracked user's body and other real objects and how irritating artifacts due to misalignments can be reduced. Our method is based on simulating the occlusion of virtual objects by a representation of the user modeled as kinematic chains of articulated solids. Registration and modeling errors of this model are being reduced by smoothing the border between virtual world and occluding real object. An implementation in our augmented environment and the resulting improvements are presented. Introduction to the problem of occlusion and related work One of the main advantages of using an augmented environment [AZ97, FE93] for collaboration as opposed to an immersive setup is the direct interaction of participants in reality. While the collaborators in an immersive setup always have to rely on more or less satisfying representations of each other, ranging from disembodied hands or heads to complete bodies visualized in plausible poses, users of an augmented scenario always are able to directly see each other and the interface devices they are using. This combination of reality and virtuality leads to the problem of correct occlusion between real and virtual objects, which of course does not exist in an immersive environment. Even when using semi-transparent HMDs, where virtual objects only appear as transparent overlays over reality, wrong occlusion can hide gestures or facial expressions of participants. When applied in a video-based augmentation setup virtual objects can completely hide real objects if not occluded properly. Even discounting the importance of the social interaction, wrongly occluding virtual objects subject the users brain to conflicting depth clues: the parallax of the stereo-rendered objects gives a farther distance than the one inferred by the occlusion of real objects. This not only leads to misconceptions of spatial properties by the user, resulting in errors when trying to grab objects, but also increases eyestrain and the probability of motion sickness. Influences of the display system As already mentioned, the properties of the display system influence the severity of the occlusion problem, as outlined in the following table: occlusion display system order (example) virtual object occluding real object real object occluding virtual object back-projection / screen based (CAVE[CR92], responsive workbench[KR94]) impossible inherent semi-transparent HMD (Studierstube [SZ98]) inherent semi-visible / software solvable video + immersive HMD (UNC [ST96b]) inherent software solvable In screen-based augmented environments fishtank scenario or projection based setups like the CAVE occlusion of virtual objects by real ones is simple and straightforward: real objects are always between the display surface and the eye and therefore always occlude virtual objects. This yields excellent results as long as no virtual object is placed between participants. The only exception for this would be the projection-based virtual office [RA98], where front projection is used. Since the virtual objects are projected on top of the real ones normally on walls or desktops real objects can be excluded from this projection by projecting nothing (displaying black) in the relevant portion of the display area. This could for example be used to exclude pictures or windows from being projected onto, but also when using the proposed optical tracking mechanism to exclude the users hands from being projected on. Nevertheless the occlusion of the display surface by real objects still manifest: users hands and arms for example may drop shadows on the surface, thereby occluding parts of objects virtually in front of the shadow. When using HMDs, the display surface is always between the eye and real objects. Without further processing virtual objects always occlude real ones. The only difference albeit only gradual exists between see-through HMDs utilizing semi-transparent mirrors and immersive HMDs being fed video images by headmounted cameras: the first only overlays semitransparent computer-generated images over reality while the second one may display completely opaque objects. Since the only case where the occlusion problem is solvable we concentrate on setups where the displayed virtual objects overlay images of reality. This can of course also be a desktopbased system, where video images are displayed overlaid by graphics. Influences of the tracking system Tracking of users (heads), input devices and real objects to be augmented is a major task which influences strongly the quality of the augmentation. Immersive environments can tolerate discrepancies between reality and computer generated images which would be impossible to ignore in an augmented setup. Since in an immersive situation the user only relies on computer generated images for hand-eye coordination, errors between hand position in reality and projected hand/cursor position in the environment almost never lead to problems when interacting with the virtual environment. In an augmented environment however, misalignment between tracked real objects and their representations in virtuality which do not have to be visual representations: an input wand may only be represented by its "hotspot", the point in space where its function takes place can cause severe problems for the user to operate in the environment. Additionally, lag between reality and the computer-generated environment is much more noticeable than in an immersive situation, since the position of virtual objects in respect to the real surroundings can be immediately compared. This results in "swimming" behavior of the virtual scene, which may also lead to motion sickness. When addressing the problem of occlusion, another quality of the tracking system used matters: the ability to supply the simulation with additional information regarding the occluding objects. Ideally, we would like the tracking system not only supply us with the position and orientation of one or more reference points on the occluding object, but also to deliver complete geometric information which enables us to determine which parts of virtual objects to occlude. This yields the following classification by tracking system: Supplies only positional data supplies additionally geometric information